Methods and apparatuses for performing artificial intelligence encoding and artificial intelligence decoding on image

ABSTRACT

Provided is an artificial intelligence (AI) decoding apparatus includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, the processor is configured to: obtain AI data related to AI down-scaling an original image to a first image; obtain image data corresponding to an encoding result on the first image; obtain a second image corresponding to the first image by performing a decoding on the image data; obtain deep neural network (DNN) setting information among a plurality of DNN setting information from the AI data; and obtain, by an up-scaling DNN, a third image by performing the AI up-scaling on the second image, the up-scaling DNN being configured with the obtained DNN setting information, wherein the plurality of DNN setting information comprises a parameter used in the up-scaling DNN, the parameter being obtained through joint training of the up-scaling DNN and a down-scaling DNN, and wherein the down-scaling DNN is used to obtain the first image from the original image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a Continuation Application of U.S. application Ser.No. 16/570,057, filed on Sep. 13, 2019, which is a ContinuationApplication of International Application No. PCT/KR2019/004171, filed onApr. 8, 2019, and claims priorities under 35 U.S.C. § 119 to KoreanPatent Application No. 10-2018-0125406, filed on Oct. 19, 2018, KoreanPatent Application No. 10-2019-0053248, filed on May 7, 2019, and KoreanPatent Application No. 10-2019-0062583, filed on May 28, 2019, thedisclosures of which are incorporated by reference herein in theirentireties.

BACKGROUND 1. Field

The disclosure relates to an image processing field. More particularly,the disclosure relates to methods and apparatuses for encoding anddecoding an image based on artificial intelligence (AI).

2. Description of Related Art

An image is stored in a recording medium or transmitted via acommunication channel in a form of a bitstream after being encoded via acodec following a certain data compression standard, such as the MovingPicture Expert Group (MPEG) standard.

With the development and supply of hardware capable of reproducing andstoring a high resolution and high quality image, the need for a codeccapable of effectively encoding and decoding the high resolution andhigh quality image has increased.

SUMMARY

Provided are methods and apparatuses for performing artificialintelligence (AI) encoding and AI decoding on an image, wherein an imageis encoded and decoded based on AI to achieve a low bitrate.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

According to an aspect of the present disclosure, an artificialintelligence (AI) decoding apparatus includes: a memory storing one ormore instructions; and a processor configured to execute the one or moreinstructions stored in the memory, wherein the processor is configuredto: obtain AI data related to AI down-scaling an original image to afirst image, the AI data comprising at least one of information relatedto the first image and information about a difference between theoriginal image and the first image; obtain image data corresponding toan encoding result on the first image; obtain a second imagecorresponding to the first image by performing a decoding on the imagedata; obtain deep neural network (DNN) setting information among aplurality of DNN setting information from the AI data, the DNN settinginformation being for performing AI up-scaling on the second image; andobtain, by an up-scaling DNN, a third image by performing the AIup-scaling on the second image, the up-scaling DNN being configured withthe obtained DNN setting information, wherein the plurality of DNNsetting information comprises a parameter used in the up-scaling DNN,the parameter being obtained through joint training of the up-scalingDNN and a down-scaling DNN, and wherein the down-scaling DNN is used toobtain the first image from the original image.

The AI data may include the information about the difference between theoriginal image and the first image, and the processor may be furtherconfigured to obtain the DNN setting information for performing AIup-scaling on the second image to match the third image with thedifference between the original image and the first image.

The AI data may include the information related to the first image, andthe processor may be further configured to obtain the DNN settinginformation mapped to the information related to the first image, basedon a mapping relationship between several image-related information andthe plurality of DNN setting information, wherein the informationrelated to the first image comprises at least one of a resolution, abitrate or a codec type.

The image data may include quantization parameter information used inthe decoding, and the processor may be further configured to obtain,based on the quantization parameter information and the informationrelated to the first image, the DNN setting information.

The obtained DNN setting information may include parameters of a filterkernel, the filter kernel may be associated with at least oneconvolution layer, and the up-scaling DNN may include the at least oneconvolution layer.

The processor may be further configured to set the up-scaling DNN withthe obtained DNN setting information instead of DNN setting informationset in the up-scaling DNN, when the DNN setting information set in theup-scaling DNN is different from the obtained DNN setting information.

The up-scaling DNN may be trained based on quality loss information,wherein the quality loss information may be correspond to a comparisonof a training image output from the up-scaling DNN and an originaltraining image before AI down-scaling is performed.

The quality loss information may be used in training of the down-scalingDNN.

When parameters of a DNN of any one of the up-scaling DNN and thedown-scaling DNN are updated during a training process, parameters of aDNN of the other one may be updated.

According to another aspect of the present disclosure, a systemincludes: an AI encoding apparatus including a down-scaling DNN, thedown-scaling DNN being configured to be trained based on: structuralloss information corresponding to a comparison of a first training imageoutput from the down-scaling DNN and a reduced training image,complexity loss information corresponding to a spatial complexity of thefirst training image, and quality loss information corresponding to acomparison of the original training image and a third training imageoutput from the up-scaling DNN; and an AI decoding apparatus includingthe up-scaling DNN, the up-scaling DNN being configured to be trainedbased on the quality loss information.

According to another aspect of the present disclosure, an artificialintelligence (AI) encoding apparatus includes: a memory storing one ormore instructions; and a processor configured to execute the one or moreinstructions stored in the memory to: obtain, by a down-scaling deepneural network (DNN) performing AI down-scaling on an original image, afirst image, the down-scaling DNN being configured with DNN settinginformation; encode the first image to obtain image data; and transmitthe image data and AI data for selecting DNN setting information of anup-scaling DNN, the AI data comprising at least one of informationrelated to the first image and information about a difference betweenthe original image and the first image, wherein the up-scaling DNN isconfigured to perform AI up-scaling on a second image, wherein thesecond image is obtained by decoding the image data, and wherein the DNNsetting information of the down-scaling DNN and the DNN settinginformation of the up-scaling DNN comprises a parameter used in thedown-scaling DNN and the up-scaling DNN, the parameter being obtainedthrough joint training of the up-scaling DNN and a down-scaling DNN.

The down-scaling DNN and the up-scaling DNN may be trained based onquality loss information, wherein the quality loss information may becorrespond to a comparison of a training image output from theup-scaling DNN and an original training image before AI down-scaling isperformed.

According to another aspect of the present disclosure, an artificialintelligence (AI) decoding method of an image, the AI decoding methodincludes: obtaining AI data related to AI down-scaling an original imageto a first image, the AI data comprising at least one of informationrelated to the first image and information about a difference betweenthe original image and the first image; obtaining image datacorresponding to an encoding result on the first image; obtaining asecond image corresponding to the first image by performing a decodingon the image data; obtaining deep neural network (DNN) settinginformation among a plurality of DNN setting information from the AIdata, the DNN setting information being for performing AI up-scaling onthe second image; and obtaining, by an up-scaling DNN, a third image byperforming the AI up-scaling on the second image, the up-scaling DNNbeing configured with the obtained DNN setting information, wherein theplurality of DNN setting information comprises a parameter used in theup-scaling DNN, the parameter being obtained through joint training ofthe up-scaling DNN and a down-scaling DNN, and wherein the down-scalingDNN is used to obtain the first image from the original image.

According to another aspect of the present disclosure, an artificialintelligence (AI) encoding method of an image, the AI encoding methodincludes: obtaining, by a down-scaling deep neural network (DNN)performing AI down-scaling on an original image, a first image, thedown-scaling DNN being configured with DNN setting information; encodingthe first image to obtain image data; and transmitting the image dataand AI data for selecting DNN setting information of an up-scaling DNN,the AI data comprising at least one of information related to the firstimage and information about a difference between the original image andthe first image, wherein the up-scaling DNN is configured to perform AIup-scaling on a second image, wherein the second image is obtained bydecoding the image data, and wherein the DNN setting information of thedown-scaling DNN and the DNN setting information of the up-scaling DNNcomprises a parameter used in the down-scaling DNN and the up-scalingDNN, the parameter being obtained through joint training of theup-scaling DNN and a down-scaling DNN.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram for describing an artificial intelligence (AI)encoding process and an AI decoding process, according to an embodiment.

FIG. 2 is a block diagram of a configuration of an AI decoding apparatusaccording to an embodiment.

FIG. 3 is a diagram showing a second deep neural network (DNN) forperforming AI up-scaling on a second image.

FIG. 4 is a diagram for describing a convolution operation by aconvolution layer.

FIG. 5 is a table showing a mapping relationship between several piecesof image-related information and several pieces of DNN settinginformation.

FIG. 6 is a diagram showing a second image including a plurality offrames.

FIG. 7 is a block diagram of a configuration of an AI encoding apparatusaccording to an embodiment.

FIG. 8 is a diagram showing a first DNN for performing AI down-scalingon an original image.

FIG. 9 is a diagram for describing a method of training a first DNN anda second DNN.

FIG. 10 is a diagram for describing a training process of a first DNNand a second DNN by a training apparatus.

FIG. 11 is a diagram of an apparatus for performing AI down-scaling onan original image and an apparatus for performing AI up-scaling on asecond image.

FIG. 12 is a flowchart of an AI decoding method according to anembodiment.

FIG. 13 is a flowchart of an AI encoding method according to anembodiment.

DETAILED DESCRIPTION

As the disclosure allows for various changes and numerous examples,particular embodiments will be illustrated in the drawings and describedin detail in the written description. However, this is not intended tolimit the disclosure to particular modes of practice, and it will beunderstood that all changes, equivalents, and substitutes that do notdepart from the spirit and technical scope of the disclosure areencompassed in the disclosure.

In the description of embodiments, certain detailed explanations ofrelated art are omitted when it is deemed that they may unnecessarilyobscure the essence of the disclosure. Also, numbers (for example, afirst, a second, and the like) used in the description of thespecification are merely identifier codes for distinguishing one elementfrom another.

Also, in the present specification, it will be understood that whenelements are “connected” or “coupled” to each other, the elements may bedirectly connected or coupled to each other, but may alternatively beconnected or coupled to each other with an intervening elementtherebetween, unless specified otherwise.

In the present specification, regarding an element represented as a“unit” or a “module”, two or more elements may be combined into oneelement or one element may be divided into two or more elementsaccording to subdivided functions. In addition, each element describedhereinafter may additionally perform some or all of functions performedby another element, in addition to main functions of itself, and some ofthe main functions of each element may be performed entirely by anothercomponent.

Also, in the present specification, an ‘image’ or a ‘picture’ may denotea still image, a moving image including a plurality of consecutive stillimages (or frames), or a video.

Also, in the present specification, a deep neural network (DNN) is arepresentative example of an artificial neural network model simulatingbrain nerves, and is not limited to an artificial neural network modelusing a specific algorithm.

Also, in the present specification, a ‘parameter’ is a value used in anoperation process of each layer forming a neural network, and forexample, may include a weight used when an input value is applied to acertain operation expression. Here, the parameter may be expressed in amatrix form. The parameter is a value set as a result of training, andmay be updated through separate training data when necessary.

Also, in the present specification, a ‘first DNN’ indicates a DNN usedfor artificial intelligence (AI) down-scaling an image, and a ‘secondDNN’ indicates a DNN used for AI up-scaling an image.

Also, in the present specification, ‘DNN setting information’ includesinformation related to an element constituting a DNN. ‘DNN settinginformation’ includes the parameter described above as informationrelated to the element constituting the DNN. The first DNN or the secondDNN may be set by using the DNN setting information.

Also, in the present specification, an ‘original image’ denotes an imageto be an object of AI encoding, and a ‘first image’ denotes an imageobtained as a result of performing AI down-scaling on the original imageduring an AI encoding process. Also, a ‘second image’ denotes an imageobtained via first decoding during an AI decoding process, and a ‘thirdimage’ denotes an image obtained by AI up-scaling the second imageduring the AI decoding process.

Also, in the present specification, ‘AI down-scale’ denotes a process ofdecreasing resolution of an image based on AI, and ‘first encoding’denotes an encoding process according to an image compression methodbased on frequency transformation. Also, ‘first decoding’ denotes adecoding process according to an image reconstruction method based onfrequency transformation, and ‘AI up-scale’ denotes a process ofincreasing resolution of an image based on AI.

FIG. 1 is a diagram for describing an AI encoding process and an AIdecoding process, according to an embodiment.

As described above, when resolution of an image remarkably increases,the throughput of information for encoding and decoding the image isincreased, and accordingly, a method for improving efficiency ofencoding and decoding of an image is required.

As shown in FIG. 1, according to an embodiment of the disclosure, afirst image 115 is obtained by performing AI down-scaling 110 on anoriginal image 105 having high resolution. Then, first encoding 120 andfirst decoding 130 are performed on the first image 115 havingrelatively low resolution, and thus a bitrate may be largely reducedcompared to when the first encoding and the first decoding are performedon the original image 105.

In particular, in FIG. 1, the first image 115 is obtained by performingthe AI down-scaling 110 on the original image 105 and the first encoding120 is performed on the first image 115 during the AI encoding process,according to an embodiment. During the AI decoding process, AI encodingdata including AI data and image data, which are obtained as a result ofAI encoding is received, a second image 135 is obtained via the firstdecoding 130, and a third image 145 is obtained by performing AIup-scaling 140 on the second image 135.

Referring to the AI encoding process in detail, when the original image105 is received, the AI down-scaling 110 is performed on the originalimage 105 to obtain the first image 115 of certain resolution or certainquality. Here, the AI down-scaling 110 is performed based on AI, and AIfor the AI down-scaling 110 needs to be trained jointly with AI for theAI up-scaling 140 of the second image 135. This is because, when the AIfor the AI down-scaling 110 and the AI for the AI up-scaling 140 areseparately trained, a difference between the original image 105 which isan object of AI encoding and the third image 145 reconstructed throughAI decoding is increased.

In an embodiment of the disclosure, the AI data may be used to maintainsuch a joint relationship during the AI encoding process and the AIdecoding process. Accordingly, the AI data obtained through the AIencoding process may include information indicating an up-scalingtarget, and during the AI decoding process, the AI up-scaling 140 isperformed on the second image 135 according to the up-scaling targetverified based on the AI data.

The AI for the AI down-scaling 110 and the AI for the AI up-scaling 140may be embodied as a DNN. As will be described later with reference toFIG. 9, because a first DNN and a second DNN are jointly trained bysharing loss information under a certain target, an AI encodingapparatus may provide target information used during joint training ofthe first DNN and the second DNN to an AI decoding apparatus, and the AIdecoding apparatus may perform the AI up-scaling 140 on the second image135 to target resolution based on the provided target information.

Regarding the first encoding 120 and the first decoding 130 of FIG. 1,information amount of the first image 115 obtained by performing AIdown-scaling 110 on the original image 105 may be reduced through thefirst encoding 120. The first encoding 120 may include a process ofgenerating prediction data by predicting the first image 115, a processof generating residual data corresponding to a difference between thefirst image 115 and the prediction data, a process of transforming theresidual data of a spatial domain component to a frequency domaincomponent, a process of quantizing the residual data transformed to thefrequency domain component, and a process of entropy-encoding thequantized residual data. Such first encoding 120 may be performed viaone of image compression methods using frequency transformation, such asMPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency VideoCoding (HEVC), VC-1, VP8, VP9, and AOMedia Video 1 (AV1).

The second image 135 corresponding to the first image 115 may bereconstructed by performing the first decoding 130 on the image data.The first decoding 130 may include a process of generating the quantizedresidual data by entropy-decoding the image data, a process ofinverse-quantizing the quantized residual data, a process oftransforming the residual data of the frequency domain component to thespatial domain component, a process of generating the prediction data,and a process of reconstructing the second image 135 by using theprediction data and the residual data. Such first decoding 130 may beperformed via an image reconstruction method corresponding to one ofimage compression methods using frequency transformation, such asMPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1, which is usedin the first encoding 120.

The AI encoding data obtained through the AI encoding process mayinclude the image data obtained as a result of performing the firstencoding 120 on the first image 115, and the AI data related to the AIdown-scaling 110 of the original image 105. The image data may be usedduring the first decoding 130 and the AI data may be used during the AIup-scaling 140.

The image data may be transmitted in a form of a bitstream. The imagedata may include data obtained based on pixel values in the first image115, for example, residual data that is a difference between the firstimage 115 and prediction data of the first image 115. Also, the imagedata includes information used during the first encoding 120 performedon the first image 115. For example, the image data may includeprediction mode information, motion information, and information relatedto quantization parameter used during the first encoding 120. The imagedata may be generated according to a rule, for example, according to asyntax, of an image compression method used during the first encoding120, among MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1.

The AI data is used in the AI up-scaling 140 based on the second DNN. Asdescribed above, because the first DNN and the second DNN are jointlytrained, the AI data includes information enabling the AI up-scaling 140to be performed accurately on the second image 135 through the secondDNN. During the AI decoding process, the AI up-scaling 140 may beperformed on the second image 135 to have targeted resolution and/orquality, based on the AI data.

The AI data may be transmitted together with the image data in a form ofa bitstream. Alternatively, according to an embodiment, the AI data maybe transmitted separately from the image data, in a form of a frame or apacket. The AI data and the image data obtained as a result of the AIencoding may be transmitted through the same network or throughdifferent networks.

FIG. 2 is a block diagram of a configuration of an AI decoding apparatus100 according to an embodiment.

Referring to FIG. 2, the AI decoding apparatus 200 according to anembodiment may include a receiver 210 and an AI decoder 230. Thereceiver 210 may include a communicator 212, a parser 214, and anoutputter 216. The AI decoder 230 may include a first decoder 232 and anAI up-scaler 234.

The receiver 210 receives and parses AI encoding data obtained as aresult of AI encoding, and distinguishably outputs image data and AIdata to the AI decoder 230.

In particular, the communicator 212 receives the AI encoding dataobtained as the result of AI encoding through a network. The AI encodingdata obtained as the result of performing AI encoding includes the imagedata and the AI data. The image data and the AI data may be receivedthrough a same type of network or different types of networks.

The parser 214 receives the AI encoding data received through thecommunicator 212 and parses the AI encoding data to distinguish theimage data and the AI data. For example, the parser 214 may distinguishthe image data and the AI data by reading a header of data obtained fromthe communicator 212. According to an embodiment, the parser 214distinguishably transmits the image data and the AI data to theoutputter 216 via the header of the data received through thecommunicator 212, and the outputter 216 transmits the distinguishedimage data and AI data respectively to the first decoder 232 and the AIup-scaler 234. At this time, it may be verified that the image dataincluded in the AI encoding data is image data generated via a certaincodec (for example, MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, orAV1). In this case, corresponding information may be transmitted to thefirst decoder 232 through the outputter 216 such that the image data isprocessed via the verified codec.

According to an embodiment, the AI encoding data parsed by the parser214 may be obtained from a data storage medium including a magneticmedium such as a hard disk, a floppy disk, or a magnetic tape, anoptical recording medium such as CD-ROM or DVD, or a magneto-opticalmedium such as a floptical disk.

The first decoder 232 reconstructs the second image 135 corresponding tothe first image 115, based on the image data. The second image 135obtained by the first decoder 232 is provided to the AI up-scaler 234.According to an embodiment, first decoding related information, such asprediction mode information, motion information, quantization parameterinformation, or the like included in the image data may be furtherprovided to the AI up-scaler 234.

Upon receiving the AI data, the AI up-scaler 234 performs AI up-scalingon the second image 135, based on the AI data. According to anembodiment, the AI up-scaling may be performed by further using thefirst decoding related information, such as the prediction modeinformation, the quantization parameter information, or the likeincluded in the image data.

The receiver 210 and the AI decoder 230 according to an embodiment aredescribed as individual devices, but may be implemented through oneprocessor. In this case, the receiver 210 and the AI decoder 230 may beimplemented through an dedicated processor or through a combination ofsoftware and general-purpose processor such as application processor(AP), central processing unit (CPU) or graphic processing unit (GPU).The dedicated processor may be implemented by including a memory forimplementing an embodiment of the disclosure or by including a memoryprocessor for using an external memory.

Also, the receiver 210 and the AI decoder 230 may be configured by aplurality of processors. In this case, the receiver 210 and the AIdecoder 230 may be implemented through a combination of dedicatedprocessors or through a combination of software and general-purposeprocessors such as AP, CPU or GPU. Similarly, the AI up-scaler 234 andthe first decoder 232 may be implemented by different processors.

The AI data provided to the AI up-scaler 234 includes informationenabling the second image 135 to be processed via AI up-scaling. Here,an up-scaling target should correspond to down-scaling of a first DNN.Accordingly, the AI data includes information for verifying adown-scaling target of the first DNN.

Examples of the information included in the AI data include differenceinformation between resolution of the original image 105 and resolutionof the first image 115, and information related to the first image 115.

The difference information may be expressed as information about aresolution conversion degree of the first image 115 compared to theoriginal image 105 (for example, resolution conversion rateinformation). Also, because the resolution of the first image 115 isverified through the resolution of the reconstructed second image 135and the resolution conversion degree is verified accordingly, thedifference information may be expressed only as resolution informationof the original image 105. Here, the resolution information may beexpressed as vertical/horizontal sizes or as a ratio (16:9, 4:3, or thelike) and a size of one axis. Also, when there is pre-set resolutioninformation, the resolution information may be expressed in a form of anindex or flag.

The information related to the first image 115 may include informationabout at least one of a bitrate of the image data obtained as the resultof performing first encoding on the first image 115 or a codec type usedduring the first encoding of the first image 115.

The AI up-scaler 234 may determine the up-scaling target of the secondimage 135, based on at least one of the difference information or theinformation related to the first image 115, which are included in the AIdata. The up-scaling target may indicate, for example, to what degreeresolution is to be up-scaled for the second image 135. When theup-scaling target is determined, the AI up-scaler 234 performs AIup-scaling on the second image 135 through a second DNN to obtain thethird image 145 corresponding to the up-scaling target.

Before describing a method, performed by the AI up-scaler 234, ofperforming AI up-scaling on the second image 135 according to theup-scaling target, an AI up-scaling process through the second DNN willbe described with reference to FIGS. 3 and 4.

FIG. 3 is a diagram showing a second DNN 300 for performing AIup-scaling on the second image 135, and FIG. 4 is a diagram fordescribing a convolution operation in a first convolution layer 310 ofFIG. 3.

As shown in FIG. 3, the second image 135 is input to the firstconvolution layer 310. 3×3×4 indicated in the first convolution layer310 shown in FIG. 3 indicates that a convolution process is performed onone input image by using four filter kernels having a size of 3×3. Fourfeature maps are generated by the four filter kernels as a result of theconvolution process. Each feature map indicates inherent characteristicsof the second image 135. For example, each feature map may represent avertical direction characteristic, a horizontal directioncharacteristic, or an edge characteristic, etc of the second image 135.

A convolution operation in the first convolution layer 310 will bedescribed in detail with reference to FIG. 4.

One feature map 450 may be generated through multiplication and additionbetween parameters of a filter kernel 430 having a size of 3×3 used inthe first convolution layer 310 and corresponding pixel values in thesecond image 135. Because four filter kernels are used in the firstconvolution layer 310, four feature maps may be generated through theconvolution operation using the four filter kernels.

I1 through I49 indicated in the second image 135 in FIG. 4 indicatepixels in the second image 135, and F1 through F9 indicated in thefilter kernel 430 indicate parameters of the filter kernel 430. Also, M1through M9 indicated in the feature map 450 indicate samples of thefeature map 450.

In FIG. 4, the second image 135 includes 49 pixels, but the number ofpixels is only an example and when the second image 135 has a resolutionof 4 K, the second image 135 may include, for example, 3840×2160 pixels.

During a convolution operation process, pixel values of I1, I2, I3, I8,I9, I10, I15, I16, and I17 of the second image 135 and F1 through F9 ofthe filter kernels 430 are respectively multiplied, and a value ofcombination (for example, addition) of result values of themultiplication may be assigned as a value of M1 of the feature map 450.When a stride of the convolution operation is 2, pixel values of I3, I4,I5, I10, I11, I12, I17, I18, and I19 of the second image 135 and F1through F9 of the filter kernels 430 are respectively multiplied, andthe value of the combination of the result values of the multiplicationmay be assigned as a value of M2 of the feature map 450.

While the filter kernel 430 moves along the stride to the last pixel ofthe second image 135, the convolution operation is performed between thepixel values in the second image 135 and the parameters of the filterkernel 430, and thus the feature map 450 having a certain size may begenerated.

According to the present disclosure, values of parameters of a secondDNN, for example, values of parameters of a filter kernel used inconvolution layers of the second DNN (for example, F1 through F9 of thefilter kernel 430), may be optimized through joint training of a firstDNN and the second DNN. As described above, the AI up-scaler 234 maydetermine an up-scaling target corresponding to a down-scaling target ofthe first DNN based on AI data, and determine parameters correspondingto the determined up-scaling target as the parameters of the filterkernel used in the convolution layers of the second DNN.

Convolution layers included in the first DNN and the second DNN mayperform processes according to the convolution operation processdescribed with reference to FIG. 4, but the convolution operationprocess described with reference to FIG. 4 is only an example and is notlimited thereto.

Referring back to FIG. 3, the feature maps output from the firstconvolution layer 310 may be input to a first activation layer 320.

The first activation layer 320 may assign a non-linear feature to eachfeature map. The first activation layer 320 may include a sigmoidfunction, a Tan h function, a rectified linear unit (ReLU) function, orthe like, but is not limited thereto.

The first activation layer 320 assigning the non-linear featureindicates that at least one sample value of the feature map, which is anoutput of the first convolution layer 310, is changed. Here, the changeis performed by applying the non-linear feature.

The first activation layer 320 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 310to the second convolution layer 330. For example, some of the samplevalues of the feature maps are activated by the first activation layer320 and transmitted to the second convolution layer 330, and some of thesample values are deactivated by the first activation layer 320 and nottransmitted to the second convolution layer 330. The intrinsiccharacteristics of the second image 135 represented by the feature mapsare emphasized by the first activation layer 320.

Feature maps 325 output from the first activation layer 320 are input tothe second convolution layer 330. One of the feature maps 325 shown inFIG. 3 is a result of processing the feature map 450 described withreference to FIG. 4 in the first activation layer 320.

3×3×4 indicated in the second convolution layer 330 indicates that aconvolution process is performed on the feature maps 325 by using fourfilter kernels having a size of 3×3. An output of the second convolutionlayer 330 is input to a second activation layer 340. The secondactivation layer 340 may assign a non-linear feature to input data.

Feature maps 345 output from the second activation layer 340 are inputto a third convolution layer 350. 3×3×1 indicated in the thirdconvolution layer 350 shown in FIG. 3 indicates that a convolutionprocess is performed to generate one output image by using one filterkernel having a size of 3×3. The third convolution layer 350 is a layerfor outputting a final image and generates one output by using onefilter kernel. According to an embodiment of the disclosure, the thirdconvolution layer 350 may output the third image 145 as a result of aconvolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 310, 330, and 350 of the second DNN 300, a parameterof filter kernels of the first, second, and third convolution layers310, 330, and 350 of the second DNN 300, and the like, as will bedescribed later, and the plurality of pieces of DNN setting informationshould be connected to a plurality of pieces of DNN setting informationof a first DNN. The connection between the plurality of pieces of DNNsetting information of the second DNN and the plurality of pieces of DNNsetting information of the first DNN may be realized via joint trainingof the first DNN and the second DNN.

In FIG. 3, the second DNN 300 includes three convolution layers (thefirst, second, and third convolution layers 310, 330, and 350) and twoactivation layers (the first and second activation layers 320 and 340),but this is only an example, and the numbers of convolution layers andactivation layers may vary according to an embodiment. Also, accordingto an embodiment, the second DNN 300 may be implemented as a recurrentneural network (RNN). In this case, a convolutional neural network (CNN)structure of the second DNN 300 according to an embodiment of thedisclosure is changed to an RNN structure.

According to an embodiment, the AI up-scaler 234 may include at leastone arithmetic logic unit (ALU) for the convolution operation and theoperation of the activation layer described above. The ALU may beimplemented as a processor. For the convolution operation, the ALU mayinclude a multiplier that performs multiplication between sample valuesof the second image 135 or the feature map output from previous layerand sample values of the filter kernel, and an adder that adds resultvalues of the multiplication. Also, for the operation of the activationlayer, the ALU may include a multiplier that multiplies an input samplevalue by a weight used in a pre-determined sigmoid function, a Tan hfunction, or an ReLU function, and a comparator that compares amultiplication result and a certain value to determine whether totransmit the input sample value to a next layer.

Hereinafter, a method, performed by the AI up-scaler 234, of performingthe AI up-scaling on the second image 135 according to the up-scalingtarget will be described.

According to an embodiment, the AI up-scaler 234 may store a pluralityof pieces of DNN setting information settable in a second DNN.

Here, the DNN setting information may include information about at leastone of the number of convolution layers included in the second DNN, thenumber of filter kernels for each convolution layer, or a parameter ofeach filter kernel. The plurality of pieces of DNN setting informationmay respectively correspond to various up-scaling targets, and thesecond DNN may operate based on DNN setting information corresponding toa certain up-scaling target. The second DNN may have differentstructures based on the DNN setting information. For example, the secondDNN may include three convolution layers based on any piece of DNNsetting information, and may include four convolution layers based onanother piece of DNN setting information.

According to an embodiment, the DNN setting information may only includea parameter of a filter kernel used in the second DNN. In this case, thestructure of the second DNN does not change, but only the parameter ofthe internal filter kernel may change based on the DNN settinginformation.

The AI up-scaler 234 may obtain the DNN setting information forperforming AI up-scaling on the second image 135, among the plurality ofpieces of DNN setting information. Each of the plurality of pieces ofDNN setting information used at this time is information for obtainingthe third image 145 of pre-determined resolution and/or pre-determinedquality, and is trained jointly with a first DNN.

For example, one piece of DNN setting information among the plurality ofpieces of DNN setting information may include information for obtainingthe third image 145 of resolution twice higher than resolution of thesecond image 135, for example, the third image 145 of 4 K (4096×2160)twice higher than 2 K (2048×1080) of the second image 135, and anotherpiece of DNN setting information may include information for obtainingthe third image 145 of resolution four times higher than the resolutionof the second image 135, for example, the third image 145 of 8 K(8192×4320) four times higher than 2 K (2048×1080) of the second image135.

Each of the plurality of pieces of DNN setting information is obtainedjointly with DNN setting information of the first DNN of an AI encodingapparatus 600 of FIG. 6, and the AI up-scaler 234 obtains one piece ofDNN setting information among the plurality of pieces of DNN settinginformation according to an enlargement ratio corresponding to areduction ratio of the DNN setting information of the first DNN. In thisregard, the AI up-scaler 234 may verify information of the first DNN. Inorder for the AI up-scaler 234 to verify the information of the firstDNN, the AI decoding apparatus 200 according to an embodiment receivesAI data including the information of the first DNN from the AI encodingapparatus 600.

In other words, the AI up-scaler 234 may verify information targeted byDNN setting information of the first DNN used to obtain the first image115 and obtain the DNN setting information of the second DNN trainedjointly with the DNN setting information of the first DNN, by usinginformation received from the AI encoding apparatus 600.

When DNN setting information for performing the AI up-scaling on thesecond image 135 is obtained from among the plurality of pieces of DNNsetting information, input data may be processed based on the second DNNoperating according to the obtained DNN setting information.

For example, when any one piece of DNN setting information is obtained,the number of filter kernels included in each of the first, second, andthird convolution layers 310, 330, and 350 of the second DNN 300 of FIG.3, and the parameters of the filter kernels are set to values includedin the obtained DNN setting information.

In particular, parameters of a filter kernel of 3×3 used in any oneconvolution layer of the second DNN of FIG. 4 are set to {1, 1, 1, 1, 1,1, 1, 1, 1}, and when DNN setting information is changed afterwards, theparameters are replaced by {2, 2, 2, 2, 2, 2, 2, 2, 2} that areparameters included in the changed DNN setting information.

The AI up-scaler 234 may obtain the DNN setting information for AIup-scaling from among the plurality of pieces of DNN settinginformation, based on information included in the AI data, and the AIdata used to obtain the DNN setting information will now be described.

According to an embodiment, the AI up-scaler 234 may obtain the DNNsetting information for AI up-scaling from among the plurality of piecesof DNN setting information, based on difference information included inthe AI data. For example, when it is verified that the resolution (forexample, 4 K (4096×2160)) of the original image 105 is twice higher thanthe resolution (for example, 2 K (2048×1080)) of the first image 115,based on the difference information, the AI up-scaler 234 may obtain theDNN setting information for increasing the resolution of the secondimage 135 two times.

According to another embodiment, the AI up-scaler 234 may obtain the DNNsetting information for AI up-scaling the second image 135 from amongthe plurality of pieces of DNN setting information, based on informationrelated to the first image 115 included in the AI data. The AI up-scaler234 may pre-determine a mapping relationship between image-relatedinformation and DNN setting information, and obtain the DNN settinginformation mapped to the information related to the first image 115.

FIG. 5 is a table showing a mapping relationship between several piecesof image-related information and several pieces of DNN settinginformation.

Through an embodiment according to FIG. 5, it will be determined that AIencoding and AI decoding processes according to an embodiment of thedisclosure do not only consider a change of resolution. As shown in FIG.5, DNN setting information may be selected considering resolution, suchas standard definition (SD), high definition (HD), or full HD, abitrate, such as 10 Mbps, 15 Mbps, or 20 Mbps, and codec information,such as AV1, H.264, or HEVC, individually or collectively. For suchconsideration of the resolution, the bitrate and the codec information,training in consideration of each element should be jointly performedwith encoding and decoding processes during an AI training process (seeFIG. 9).

Accordingly, when a plurality of pieces of DNN setting information areprovided based on image-related information including a codec type,resolution of an image, and the like, as shown in FIG. 5 according totraining, the DNN setting information for AI up-scaling the second image135 may be obtained based on the information related to the first image115 received during the AI decoding process.

In other words, the AI up-scaler 234 is capable of using DNN settinginformation according to image-related information by matching theimage-related information at the left of a table of FIG. 5 and the DNNsetting information at the right of the table.

As shown in FIG. 5, when it is verified, from the information related tothe first image 115, that the resolution of the first image 115 is SD, abitrate of image data obtained as a result of performing first encodingon the first image 115 is 10 Mbps, and the first encoding is performedon the first image 115 via AV1 codec, the AI up-scaler 234 may use A DNNsetting information among the plurality of pieces of DNN settinginformation.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is HD, the bitrateof the image data obtained as the result of performing the firstencoding is 15 Mbps, and the first encoding is performed via H.264codec, the AI up-scaler 234 may use B DNN setting information among theplurality of pieces of DNN setting information.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is full HD, thebitrate of the image data obtained as the result of performing the firstencoding is 20 Mbps, and the first encoding is performed via HEVC codec,the AI up-scaler 234 may use C DNN setting information among theplurality of pieces of DNN setting information, and when it is verifiedthat the resolution of the first image 115 is full HD, the bitrate ofthe image data obtained as the result of performing the first encodingis 15 Mbps, and the first encoding is performed via HEVC codec, the AIup-scaler 234 may use D DNN setting information among the plurality ofpieces of DNN setting information. One of the C DNN setting informationand the D DNN setting information is selected based on whether thebitrate of the image data obtained as the result of performing the firstencoding on the first image 115 is 20 Mbps or 15 Mbps. The differentbitrates of the image data, obtained when the first encoding isperformed on the first image 115 of the same resolution via the samecodec, indicates different qualities of reconstructed images.Accordingly, a first DNN and a second DNN may be jointly trained basedon certain image quality, and accordingly, the AI up-scaler 234 mayobtain DNN setting information according to a bitrate of image dataindicating the quality of the second image 135.

According to another embodiment, the AI up-scaler 234 may obtain the DNNsetting information for performing AI up-scaling on the second image 135from among the plurality of pieces of DNN setting informationconsidering both information (prediction mode information, motioninformation, quantization parameter information, and the like) providedfrom the first decoder 232 and the information related to the firstimage 115 included in the AI data. For example, the AI up-scaler 234 mayreceive quantization parameter information used during a first encodingprocess of the first image 115 from the first decoder 232, verify abitrate of image data obtained as an encoding result of the first image115 from AI data, and obtain DNN setting information corresponding tothe quantization parameter information and the bitrate. Even when thebitrates are the same, the quality of reconstructed images may varyaccording to the complexity of an image. A bitrate is a valuerepresenting the entire first image 115 on which first encoding isperformed, and the quality of each frame may vary even within the firstimage 115. Accordingly, DNN setting information more suitable for thesecond image 135 may be obtained when prediction mode information,motion information, and/or a quantization parameter obtainable for eachframe from the first decoder 232 are/is considered together, compared towhen only the AI data is used.

Also, according to an embodiment, the AI data may include an identifierof mutually agreed DNN setting information. An identifier of DNN settinginformation is information for distinguishing a pair of pieces of DNNsetting information jointly trained between the first DNN and the secondDNN, such that AI up-scaling is performed on the second image 135 to theup-scaling target corresponding to the down-scaling target of the firstDNN. The AI up-scaler 234 may perform AI up-scaling on the second image135 by using the DNN setting information corresponding to the identifierof the DNN setting information, after obtaining the identifier of theDNN setting information included in the AI data. For example,identifiers indicating each of the plurality of DNN setting informationsettable in the first DNN and identifiers indicating each of theplurality of DNN setting information settable in the second DNN may bepreviously designated. In this case, the same identifier may bedesignated for a pair of DNN setting information settable in each of thefirst DNN and the second DNN. The AI data may include an identifier ofDNN setting information set in the first DNN for AI down-scaling of theoriginal image 105. The AI up-scaler 234 that receives the AI data mayperform AI up-scaling on the second image 135 by using the DNN settinginformation indicated by the identifier included in the AI data amongthe plurality of DNN setting information.

Also, according to an embodiment, the AI data may include the DNNsetting information. The AI up-scaler 234 may perform AI up-scaling onthe second image 135 by using the DNN setting information afterobtaining the DNN setting information included in the AI data.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the AI up-scaler 234 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on information included in the AI data, and perform AIup-scaling on the second image 135 by using the obtained DNN settinginformation.

According to an embodiment, when a structure of DNN corresponding to theup-scaling target is determined, the AI up-scaler 234 may obtain the DNNsetting information, for example, parameters of a filter kernel,corresponding to the determined structure of DNN.

The AI up-scaler 234 obtains the DNN setting information of the secondDNN through the AI data including information related to the first DNN,and performs AI up-scaling on the second image 135 through the secondDNN set based on the obtained DNN setting information, and in this case,memory usage and throughput may be reduced compared to when features ofthe second image 135 are directly analyzed for up-scaling.

According to an embodiment, when the second image 135 includes aplurality of frames, the AI up-scaler 234 may independently obtain DNNsetting information for a certain number of frames, or may obtain commonDNN setting information for entire frames.

FIG. 6 is a diagram showing the second image 135 including a pluralityof frames.

As shown in FIG. 6, the second image 135 may include frames t0 throughtn.

According to an embodiment, the AI up-scaler 234 may obtain DNN settinginformation of a second DNN through AI data, and perform AI up-scalingon the frames t0 through tn based on the obtained DNN settinginformation. In other words, the frames t0 through tn may be processedvia AI up-scaling based on common DNN setting information.

According to another embodiment, the AI up-scaler 234 may perform AIup-scaling on some of the frames t0 through tn, for example, the framest0 through ta, by using ‘A’ DNN setting information obtained from AIdata, and perform AI up-scaling on the frames ta+1 through tb by using‘B’ DNN setting information obtained from the AI data. Also, the AIup-scaler 234 may perform AI up-scaling on the frames tb+1 through tn byusing ‘C’ DNN setting information obtained from the AI data. In otherwords, the AI up-scaler 234 may independently obtain DNN settinginformation for each group including a certain number of frames amongthe plurality of frames, and perform AI up-scaling on frames included ineach group by using the independently obtained DNN setting information.

According to another embodiment, the AI up-scaler 234 may independentlyobtain DNN setting information for each frame forming the second image135. In other words, when the second image 135 includes three frames,the AI up-scaler 234 may perform AI up-scaling on a first frame by usingDNN setting information obtained in relation to the first frame, performAI up-scaling on a second frame by using DNN setting informationobtained in relation to the second frame, and perform AI up-scaling on athird frame by using DNN setting information obtained in relation to thethird frame. DNN setting information may be independently obtained foreach frame included in the second image 135, according to a method ofobtaining DNN setting information based on information (prediction modeinformation, motion information, quantization parameter information, orthe like) provided from the first decoder 232 and information related tothe first image 115 included in the AI data described above. This isbecause the mode information, the quantization parameter information, orthe like may be determined independently for each frame included in thesecond image 135.

According to another embodiment, the AI data may include informationabout to which frame DNN setting information obtained based on the AIdata is valid. For example, when the AI data includes informationindicating that DNN setting information is valid up to the frame ta, theAI up-scaler 234 performs AI up-scaling on the frames t0 through ta byusing DNN setting information obtained based on the AI data. Also, whenanother piece of AI data includes information indicating that DNNsetting information is valid up to the frame tn, the AI up-scaler 234performs AI up-scaling on the frames ta+1 through tn by using DNNsetting information obtained based on the other piece of AI data.

Hereinafter, the AI encoding apparatus 600 for performing AI encoding onthe original image 105 will be described with reference to FIG. 7.

FIG. 7 is a block diagram of a configuration of the AI encodingapparatus 600 according to an embodiment.

Referring to FIG. 7, the AI encoding apparatus 600 may include an AIencoder 610 and a transmitter 630. The AI encoder 610 may include an AIdown-scaler 612 and a first encoder 614. The transmitter 630 may includea data processor 632 and a communicator 634.

In FIG. 7, the AI encoder 610 and the transmitter 630 are illustrated asseparate devices, but the AI encoder 610 and the transmitter 630 may beimplemented through one processor. In this case, the AI encoder 610 andthe transmitter 630 may be implemented through an dedicated processor orthrough a combination of software and general-purpose processor such asAP, CPU or graphics processing unit GPU. The dedicated processor may beimplemented by including a memory for implementing an embodiment of thedisclosure or by including a memory processor for using an externalmemory.

Also, the AI encoder 610 and the transmitter 630 may be configured by aplurality of processors. In this case, the AI encoder 610 and thetransmitter 630 may be implemented through a combination of dedicatedprocessors or through a combination of software and a plurality ofgeneral-purpose processors such as AP, CPU or GPU. The AI down-scaler612 and the first encoder 614 may be implemented through differentprocessors.

The AI encoder 610 performs AI down-scaling on the original image 105and first encoding on the first image 115, and transmits AI data andimage data to the transmitter 630. The transmitter 630 transmits the AIdata and the image data to the AI decoding apparatus 200.

The image data includes data obtained as a result of performing thefirst encoding on the first image 115. The image data may include dataobtained based on pixel values in the first image 115, for example,residual data that is a difference between the first image 115 andprediction data of the first image 115. Also, the image data includesinformation used during a first encoding process of the first image 115.For example, the image data may include prediction mode information,motion information, quantization parameter information used to performthe first encoding on the first image 115, and the like.

The AI data includes information enabling AI up-scaling to be performedon the second image 135 to an up-scaling target corresponding to adown-scaling target of a first DNN. According to an embodiment, the AIdata may include difference information between the original image 105and the first image 115. Also, the AI data may include informationrelated to the first image 115. The information related to the firstimage 115 may include information about at least one of resolution ofthe first image 115, a bitrate of the image data obtained as the resultof performing the first encoding on the first image 115, or a codec typeused during the first encoding of the first image 115.

According to an embodiment, the AI data may include an identifier ofmutually agreed DNN setting information such that the AI up-scaling isperformed on the second image 135 to the up-scaling target correspondingto the down-scaling target of the first DNN.

Also, according to an embodiment, the AI data may include DNN settinginformation settable in a second DNN.

The AI down-scaler 612 may obtain the first image 115 obtained byperforming the AI down-scaling on the original image 105 through thefirst DNN. The AI down-scaler 612 may determine the down-scaling targetof the original image 105, based on a pre-determined standard.

In order to obtain the first image 115 matching the down-scaling target,the AI down-scaler 612 may store a plurality of pieces of DNN settinginformation settable in the first DNN. The AI down-scaler 612 obtainsDNN setting information corresponding to the down-scaling target fromamong the plurality of pieces of DNN setting information, and performsthe AI down-scaling on the original image 105 through the first DNN setin the obtained DNN setting information.

Each of the plurality of pieces of DNN setting information may betrained to obtain the first image 115 of pre-determined resolutionand/or pre-determined quality. For example, any one piece of DNN settinginformation among the plurality of pieces of DNN setting information mayinclude information for obtaining the first image 115 of resolution halfresolution of the original image 105, for example, the first image 115of 2 K (2048×1080) half 4 K (4096×2160) of the original image 105, andanother piece of DNN setting information may include information forobtaining the first image 115 of resolution quarter resolution of theoriginal image 105, for example, the first image 115 of 2 K (2048×1080)quarter 8 K (8192×4320) of the original image 105.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the AI down-scaler 612 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on the down-scaling target, and perform AI down-scaling onthe original image 105 by using the obtained DNN setting information.

According to an embodiment, the AI down-scaler 612 may determine astructure of DNN corresponding to the down-scaling target, and obtainDNN setting information corresponding to the determined structure ofDNN, for example, obtain parameters of a filter kernel.

The plurality of pieces of DNN setting information for performing the AIdown-scaling on the original image 105 may have an optimized value asthe first DNN and the second DNN are jointly trained. Here, each pieceof DNN setting information includes at least one of the number ofconvolution layers included in the first DNN, the number of filterkernels for each convolution layer, or a parameter of each filterkernel.

The AI down-scaler 612 may set the first DNN with the DNN settinginformation obtained for performing the AI down-scaling on the originalimage 105 to obtain the first image 115 of certain resolution and/orcertain quality through the first DNN. When the DNN setting informationfor performing the AI down-scaling on the original image 105 is obtainedfrom the plurality of pieces of DNN setting information, each layer inthe first DNN may process input data based on information included inthe DNN setting information.

Hereinafter, a method, performed by the AI down-scaler 612, ofdetermining the down-scaling target will be described. The down-scalingtarget may indicate, for example, by how much is resolution decreasedfrom the original image 105 to obtain the first image 115.

According to an embodiment, the AI down-scaler 612 may determine thedown-scaling target based on at least one of a compression ratio (forexample, a resolution difference between the original image 105 and thefirst image 115, target bitrate, or the like), compression quality (forexample, type of bitrate), compression history information, or a type ofthe original image 105.

For example, the AI down-scaler 612 may determine the down-scalingtarget based on the compression ratio, the compression quality, or thelike, which is pre-set or input from a user.

As another example, the AI down-scaler 612 may determine thedown-scaling target by using the compression history information storedin the AI encoding apparatus 600. For example, according to thecompression history information usable by the AI encoding apparatus 600,encoding quality, a compression ratio, or the like preferred by the usermay be determined, and the down-scaling target may be determinedaccording to the encoding quality determined based on the compressionhistory information. For example, the resolution, quality, or the likeof the first image 115 may be determined according to the encodingquality that has been used most often according to the compressionhistory information.

As another example, the AI down-scaler 612 may determine thedown-scaling target based on the encoding quality that has been usedmore frequently than a certain threshold value (for example, averagequality of the encoding quality that has been used more frequently thanthe certain threshold value), according to the compression historyinformation.

As another example, the AI down-scaler 612 may determine thedown-scaling target, based on the resolution, type (for example, a fileformat), or the like of the original image 105.

According to an embodiment, when the original image 105 includes aplurality of frames, the AI down-scaler 612 may independently determinedown-scaling target for a certain number of frames, or may determinedown-scaling target for entire frames.

According to an embodiment, the AI down-scaler 612 may divide the framesincluded in the original image 105 into a certain number of groups, andindependently determine the down-scaling target for each group. The sameor different down-scaling targets may be determined for each group. Thenumber of frames included in the groups may be the same or differentaccording to the each group.

According to another embodiment, the AI down-scaler 612 mayindependently determine a down-scaling target for each frame included inthe original image 105. The same or different down-scaling targets maybe determined for each frame.

Hereinafter, an example of a structure of a first DNN 700 on which AIdown-scaling is based will be described.

FIG. 8 is a diagram showing the first DNN 700 for performing AIdown-scaling on the original image 105.

As shown in FIG. 8, the original image 105 is input to a firstconvolution layer 710. The first convolution layer 710 performs aconvolution process on the original image 105 by using 32 filter kernelshaving a size of 5×5. 32 feature maps generated as a result of theconvolution process are input to a first activation layer 720. The firstactivation layer 720 may assign a non-linear feature to the 32 featuremaps.

The first activation layer 720 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 710to the second convolution layer 730. For example, some of the samplevalues of the feature maps are activated by the first activation layer720 and transmitted to the second convolution layer 730, and some of thesample values are deactivated by the first activation layer 720 and nottransmitted to the second convolution layer 730. Information representedby the feature maps output from the first convolution layer 710 isemphasized by the first activation layer 720.

An output 725 of the first activation layer 720 is input to a secondconvolution layer 730. The second convolution layer 730 performs aconvolution process on input data by using 32 filter kernels having asize of 5×5. 32 feature maps output as a result of the convolutionprocess are input to a second activation layer 740, and the secondactivation layer 740 may assign a non-linear feature to the 32 featuremaps.

An output 745 of the second activation layer 740 is input to a thirdconvolution layer 750. The third convolution layer 750 performs aconvolution process on input data by using one filter kernel having asize of 5×5. As a result of the convolution process, one image may beoutput from the third convolution layer 750. The third convolution layer750 generates one output by using the one filter kernel as a layer foroutputting a final image. According to an embodiment of the disclosure,the third convolution layer 750 may output the first image 115 as aresult of a convolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 710, 730, and 750 of the first DNN 700, a parameterof each filter kernel of the first, second, and third convolution layers710, 730, and 750 of the first DNN 700, and the like, and the pluralityof pieces of DNN setting information may be connected to a plurality ofpieces of DNN setting information of a second DNN. The connectionbetween the plurality of pieces of DNN setting information of the firstDNN and the plurality of pieces of DNN setting information of the secondDNN may be realized via joint training of the first DNN and the secondDNN.

In FIG. 8, the first DNN 700 includes three convolution layers (thefirst, second, and third convolution layers 710, 730, and 750) and twoactivation layers (the first and second activation layers 720 and 740),but this is only an example, and the numbers of convolution layers andactivation layers may vary according to an embodiment. Also, accordingto an embodiment, the first DNN 700 may be implemented as an RNN. Inthis case, a CNN structure of the first DNN 700 according to anembodiment of the disclosure is changed to an RNN structure.

According to an embodiment, the AI down-scaler 612 may include at leastone ALU for the convolution operation and the operation of theactivation layer described above. The ALU may be implemented as aprocessor. For the convolution operation, the ALU may include amultiplier that performs multiplication between sample values of theoriginal image 105 or the feature map output from previous layer andsample values of the filter kernel, and an adder that adds result valuesof the multiplication. Also, for the operation of the activation layer,the ALU may include a multiplier that multiplies an input sample valueby a weight used in a pre-determined sigmoid function, a Tan h function,or an ReLU function, and a comparator that compares a multiplicationresult and a certain value to determine whether to transmit the inputsample value to a next layer.

Referring back to FIG. 7, upon receiving the first image 115 from the AIdown-scaler 612, the first encoder 614 may reduce an information amountof the first image 115 by performing first encoding on the first image115. The image data corresponding to the first image 115 may be obtainedas a result of performing the first encoding by the first encoder 614.

The data processor 632 processes at least one of the AI data or theimage data to be transmitted in a certain form. For example, when the AIdata and the image data are to be transmitted in a form of a bitstream,the data processor 632 may process the AI data to be expressed in a formof a bitstream, and transmit the image data and the AI data in a form ofone bitstream through the communicator 634. As another example, the dataprocessor 632 may process the AI data to be expressed in a form ofbitstream, and transmit each of a bitstream corresponding to the AI dataand a bitstream corresponding to the image data through the communicator634. As another example, the data processor 632 may process the AI datato be expressed in a form of a frame or packet, and transmit the imagedata in a form of a bitstream and the AI data in a form of a frame orpacket through the communicator 634.

The communicator 634 transmits AI encoding data obtained as a result ofperforming AI encoding, through a network. The AI encoding data obtainedas the result of performing AI encoding includes the image data and theAI data. The image data and the AI data may be transmitted through asame type of network or different types of networks.

According to an embodiment, the AI encoding data obtained as a result ofprocesses of the data processor 632 may be stored in a data storagemedium including a magnetic medium such as a hard disk, a floppy disk,or a magnetic tape, an optical recording medium such as CD-ROM or DVD,or a magneto-optical medium such as a floptical disk.

Hereinafter, a method of jointly training the first DNN 700 and thesecond DNN 300 will be described with reference to FIG. 9.

FIG. 9 is a diagram for describing a method of training the first DNN700 and the second DNN 300.

In an embodiment, the original image 105 on which AI encoding isperformed through an AI encoding process is reconstructed to the thirdimage 145 via an AI decoding process, and in order to maintainsimilarity between the original image 105 and the third image 145obtained as a result of AI decoding, connectivity is between the AIencoding process and the AI decoding process is required. In otherwords, information lost in the AI encoding process needs to bereconstructed during the AI decoding process, and in this regard, thefirst DNN 700 and the second DNN 300 need to be jointly trained.

For accurate AI decoding, ultimately, quality loss information 830corresponding to a result of comparing a third training image 804 and anoriginal training image 801 shown in FIG. 9 needs to be reduced.Accordingly, the quality loss information 830 is used to train both ofthe first DNN 700 and the second DNN 300.

First, a training process shown in FIG. 9 will be described.

In FIG. 9, the original training image 801 is an image on which AIdown-scaling is to be performed and a first training image 802 is animage obtained by performing AI down-scaling on the original trainingimage 801. Also, the third training image 804 is an image obtained byperforming AI up-scaling on the first training image 802.

The original training image 801 includes a still image or a moving imageincluding a plurality of frames. According to an embodiment, theoriginal training image 801 may include a luminance image extracted fromthe still image or the moving image including the plurality of frames.Also, according to an embodiment, the original training image 801 mayinclude a patch image extracted from the still image or the moving imageincluding the plurality of frames. When the original training image 801includes the plurality of frames, the first training image 802, thesecond training image, and the third training image 804 also eachinclude a plurality of frames. When the plurality of frames of theoriginal training image 801 are sequentially input to the first DNN 700,the plurality of frames of the first training image 802, the secondtraining image and the third training image 804 may be sequentiallyobtained through the first DNN 700 and the second DNN 300.

For joint training of the first DNN 700 and the second DNN 300, theoriginal training image 801 is input to the first DNN 700. The originaltraining image 801 input to the first DNN 700 is output as the firsttraining image 802 via the AI down-scaling, and the first training image802 is input to the second DNN 300. The third training image 804 isoutput as a result of performing the AI up-scaling on the first trainingimage 802.

Referring to FIG. 9, the first training image 802 is input to the secondDNN 300, and according to an embodiment, a second training imageobtained as first encoding and first decoding are performed on the firsttraining image 802 may be input to the second DNN 300. In order to inputthe second training image to the second DNN 300, any one codec amongMPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 may be used. Inparticular, any one codec among MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8,VP9, and AV1 may be used to perform first encoding on the first trainingimage 802 and first decoding on image data corresponding to the firsttraining image 802.

Referring to FIG. 9, separate from the first training image 802 beingoutput through the first DNN 700, a reduced training image 803 obtainedby performing legacy down-scaling on the original training image 801 isobtained. Here, the legacy down-scaling may include at least one ofbilinear scaling, bicubic scaling, lanczos scaling, or stair stepscaling.

In order to prevent a structural feature of the first image 115 fromdeviating greatly from a structural feature of the original image 105,the reduced training image 803 is obtained to preserve the structuralfeature of the original training image 801.

Before training is performed, the first DNN 700 and the second DNN 300may be set to pre-determined DNN setting information. When the trainingis performed, structural loss information 810, complexity lossinformation 820, and the quality loss information 830 may be determined.

The structural loss information 810 may be determined based on a resultof comparing the reduced training image 803 and the first training image802. For example, the structural loss information 810 may correspond toa difference between structural information of the reduced trainingimage 803 and structural information of the first training image 802.Structural information may include various features extractable from animage, such as luminance, contrast, histogram, or the like of the image.The structural loss information 810 indicates how much structuralinformation of the original training image 801 is maintained in thefirst training image 802. When the structural loss information 810 issmall, the structural information of the first training image 802 issimilar to the structural information of the original training image801.

The complexity loss information 820 may be determined based on spatialcomplexity of the first training image 802. For example, a totalvariance value of the first training image 802 may be used as thespatial complexity. The complexity loss information 820 is related to abitrate of image data obtained by performing first encoding on the firsttraining image 802. It is defined that the bitrate of the image data islow when the complexity loss information 820 is small.

The quality loss information 830 may be determined based on a result ofcomparing the original training image 801 and the third training image804. The quality loss information 830 may include at least one of anL1-norm value, an L2-norm value, an Structural Similarity (SSIM) value,a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, anMultiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF)value, or a Video Multimethod Assessment Fusion (VMAF) value regardingthe difference between the original training image 801 and the thirdtraining image 804. The quality loss information 830 indicates howsimilar the third training image 804 is to the original training image801. The third training image 804 is more similar to the originaltraining image 801 when the quality loss information 830 is small.

Referring to FIG. 9, the structural loss information 810, the complexityloss information 820 and the quality loss information 830 are used totrain the first DNN 700, and the quality loss information 830 is used totrain the second DNN 300. In other words, the quality loss information830 is used to train both the first and second DNNs 700 and 300.

The first DNN 700 may update a parameter such that final lossinformation determined based on the first through quality lossinformation 810 through 830 is reduced or minimized. Also, the secondDNN 300 may update a parameter such that the quality loss information830 is reduced or minimized.

The final loss information for training the first DNN 700 and the secondDNN 300 may be determined as Equation 1 below.LossDS=a×Structural loss information+b×Complexity lossinformation+c×Quality loss information  [Equation 1]LossUS=d×Quality loss information

In Equation 1, LossDS indicates final loss information to be reduced orminimized to train the first DNN 700, and LossUS indicates final lossinformation to be reduced or minimized to train the second DNN 300.Also, a, b, c and d may be pre-determined certain weights.

In other words, the first DNN 700 updates parameters in a directionLossDS of Equation 1 is reduced, and the second DNN 300 updatesparameters in a direction LossUS is reduced. When the parameters of thefirst DNN 700 are updated according to LossDS derived during thetraining, the first training image 802 obtained based on the updatedparameters becomes different from a previous first training image 802obtained based on not updated parameters, and accordingly, the thirdtraining image 804 also becomes different from a previous third trainingimage 804. When the third training image 804 becomes different from theprevious third training image 804, the quality loss information 830 isalso newly determined, and the second DNN 300 updates the parametersaccordingly. When the quality loss information 830 is newly determined,LossDS is also newly determined, and the first DNN 700 updates theparameters according to newly determined LossDS. In other words,updating of the parameters of the first DNN 700 leads to updating of theparameters of the second DNN 300, and updating of the parameters of thesecond DNN 300 leads to updating of the parameters of the first DNN 700.In other words, because the first DNN 700 and the second DNN 300 arejointly trained by sharing the quality loss information 830, theparameters of the first DNN 700 and the parameters of the second DNN 300may be jointly optimized.

Referring to Equation 1, it is verified that LossUS is determinedaccording to the quality loss information 830, but this is only anexample and LossUS may be determined based on at least one of thestructural loss information 810 and the complexity loss information 820,and the quality loss information 830.

Hereinabove, it has been described that the AI up-scaler 234 of the AIdecoding apparatus 200 and the AI down-scaler 612 of the AI encodingapparatus 600 store the plurality of pieces of DNN setting information,and methods of training each of the plurality of pieces of DNN settinginformation stored in the AI up-scaler 234 and the AI down-scaler 612will now be described.

As described with reference to Equation 1, the first DNN 700 updates theparameters considering the similarity (the structural loss information810) between the structural information of the first training image 802and the structural information of the original training image 801, thebitrate (the complexity loss information 820) of the image data obtainedas a result of performing first encoding on the first training image802, and the difference (the quality loss information 830) between thethird training image 804 and the original training image 801.

In particular, the parameters of the first DNN 700 may be updated suchthat the first training image 802 having similar structural informationas the original training image 801 is obtained and the image data havinga small bitrate is obtained when first encoding is performed on thefirst training image 802, and at the same time, the second DNN 300performing AI up-scaling on the first training image 802 obtains thethird training image 804 similar to the original training image 801.

A direction in which the parameters of the first DNN 700 are optimizedmay vary by adjusting the weights a, b, and c of Equation 1. Forexample, when the weight b is determined to be high, the parameters ofthe first DNN 700 may be updated by prioritizing a low bitrate over highquality of the third training image 804. Also, when the weight c isdetermined to be high, the parameters of the first DNN 700 may beupdated by prioritizing high quality of the third training image 804over a high bitrate or maintaining of the structural information of theoriginal training image 801.

Also, the direction in which the parameters of the first DNN 700 areoptimized may vary according to a type of codec used to perform firstencoding on the first training image 802. This is because the secondtraining image to be input to the second DNN 300 may vary according tothe type of codec.

In other words, the parameters of the first DNN 700 and the parametersof the second DNN 300 may be jointly updated based on the weights a, b,and c, and the type of codec for performing first encoding on the firsttraining image 802. Accordingly, when the first DNN 700 and the secondDNN 300 are trained after determining the weights a, b, and c each to acertain value and determining the type of codec to a certain type, theparameters of the first DNN 700 and the parameters of the second DNN 300connected and optimized to each other may be determined.

Also, when the first DNN 700 and the second DNN 300 are trained afterchanging the weights a, b, and c, and the type of codec, the parametersof the first DNN 700 and the parameters of the second DNN 300 connectedand optimized to each other may be determined. In other words, theplurality of pieces of DNN setting information jointly trained with eachother may be determined in the first DNN 700 and the second DNN 300 whenthe first DNN 700 and the second DNN 300 are trained while changingvalues of the weights a, b, and c, and the type of codec.

As described above with reference to FIG. 5, the plurality of pieces ofDNN setting information of the first DNN 700 and the second DNN 300 maybe mapped to the information related to the first image. To set such amapping relationship, first encoding may be performed on the firsttraining image 802 output from the first DNN 700 via a certain codecaccording to a certain bitrate and the second training image obtained byperforming first decoding on a bitstream obtained as a result ofperforming the first encoding may be input to the second DNN 300. Inother words, by training the first DNN 700 and the second DNN 300 aftersetting an environment such that the first encoding is performed on thefirst training image 802 of a certain resolution via the certain codecaccording to the certain bitrate, a DNN setting information pair mappedto the resolution of the first training image 802, a type of the codecused to perform the first encoding on the first training image 802, andthe bitrate of the bitstream obtained as a result of performing thefirst encoding on the first training image 802 may be determined. Byvariously changing the resolution of the first training image 802, thetype of codec used to perform the first encoding on the first trainingimage 802 and the bitrate of the bitstream obtained according to thefirst encoding of the first training image 802, the mappingrelationships between the plurality of DNN setting information of thefirst DNN 700 and the second DNN 300 and the pieces of informationrelated to the first image may be determined.

FIG. 10 is a diagram for describing training processes of the first DNN700 and the second DNN by a training apparatus 1000.

The training of the first DNN 700 and the second DNN 300 described withreference FIG. 9 may be performed by the training apparatus 1000. Thetraining apparatus 1000 includes the first DNN 700 and the second DNN300. The training apparatus 1000 may be, for example, the AI encodingapparatus 600 or a separate server. The DNN setting information of thesecond DNN 300 obtained as the training result is stored in the AIdecoding apparatus 200.

Referring to FIG. 10, the training apparatus 1000 initially sets the DNNsetting information of the first DNN 700 and the second DNN 300, inoperations S840 and S845. Accordingly, the first DNN 700 and the secondDNN 300 may operate according to pre-determined DNN setting information.The DNN setting information may include information about at least oneof the number of convolution layers included in the first DNN 700 andthe second DNN 300, the number of filter kernels for each convolutionlayer, the size of a filter kernel for each convolution layer, or aparameter of each filter kernel.

The training apparatus 1000 inputs the original training image 801 intothe first DNN 700, in operation S850. The original training image 801may include a still image or at least one frame included in a movingimage.

The first DNN 700 processes the original training image 801 according tothe initially set DNN setting information and outputs the first trainingimage 802 obtained by performing AI down-scaling on the originaltraining image 801, in operation S855. In FIG. 10, the first trainingimage 802 output from the first DNN 700 is directly input to the secondDNN 300, but the first training image 802 output from the first DNN 700may be input to the second DNN 300 by the training apparatus 1000. Also,the training apparatus 1000 may perform first encoding and firstdecoding on the first training image 802 via a certain codec, and theninput the second training image to the second DNN 300.

The second DNN 300 processes the first training image 802 or the secondtraining image according to the initially set DNN setting informationand outputs the third training image 804 obtained by performing AIup-scaling on the first training image 802 or the second training image,in operation S860.

The training apparatus 1000 calculates the complexity loss information820, based on the first training image 802, in operation S865.

The training apparatus 1000 calculates the structural loss information810 by comparing the reduced training image 803 and the first trainingimage 802, in operation S870.

The training apparatus 1000 calculates the quality loss information 830by comparing the original training image 801 and the third trainingimage 804, in operation S875.

The initially set DNN setting information is updated in operation S880via a back propagation process based on the final loss information. Thetraining apparatus 1000 may calculate the final loss information fortraining the first DNN 700, based on the complexity loss information820, the structural loss information 810, and the quality lossinformation 830.

The second DNN 300 updates the initially set DNN setting information inoperation S885 via a back propagation process based on the quality lossinformation 830 or the final loss information. The training apparatus1000 may calculate the final loss information for training the secondDNN 300, based on the quality loss information 830.

Then, the training apparatus 1000, the first DNN 700, and the second DNN300 may repeat operations S850 through S885 until the final lossinformation is minimized to update the DNN setting information. At thistime, during each repetition, the first DNN 700 and the second DNN 300operate according to the DNN setting information updated in the previousoperation.

Table 1 below shows effects when AI encoding and AI decoding areperformed on the original image 105 according to an embodiment of thedisclosure and when encoding and decoding are performed on the originalimage 105 via HEVC.

TABLE 1 Information Subjective Image Amount (Bitrate) Quality Score(Mbps) (VMAF) Frame Al Encoding/ Al Encoding/ Content Resolution NumberHEVC Al Decoding HEVC Al Decoding Content_01 8K 300 frames 46.3 21.494.80 93.54 Content_02 (7680 × 4320) 46.3 21.6 98.05 98.98 Content_0346.3 22.7 96.08 96.00 Content_04 46.1 22.1 86.26 92.00 Content_05 45.422.7 93.42 92.98 Content_06 46.3 23.0 95.99 95.61 Average 46.11 22.2594.10 94.85

As shown in Table 1, despite subjective image quality when AI encodingand AI decoding are performed on content including 300 frames of 8 Kresolution, according to an embodiment of the disclosure, is higher thansubjective image quality when encoding and decoding are performed viaHEVC, a bitrate is reduced by at least 50%.

FIG. 11 is a diagram of an apparatus 20 for performing AI down-scalingon the original image 105 and an apparatus 40 for performing AIup-scaling on the second image 135.

The apparatus 20 receives the original image 105 and provides image data25 and AI data 30 to the apparatus 40 by using an AI down-scaler 1124and a transformation-based encoder 1126. According to an embodiment, theimage data 25 corresponds to the image data of FIG. 1 and the AI data 30corresponds to the AI data of FIG. 1. Also, according to an embodiment,the transformation-based encoder 1126 corresponds to the first encoder614 of FIG. 7 and the AI down-scaler 1124 corresponds to the AIdown-scaler 612 of FIG. 7.

The apparatus 40 receives the AI data 30 and the image data 25 andobtains the third image 145 by using a transformation-based decoder 1146and an AI up-scaler 1144. According to an embodiment, thetransformation-based decoder 1146 corresponds to the first decoder 232of FIG. 2 and the AI up-scaler 1144 corresponds to the AI up-scaler 234of FIG. 2.

According to an embodiment, the apparatus 20 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 20performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

According to an embodiment, the apparatus 40 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 40performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

In FIG. 11, a configuration controller 1122 receives at least one inputvalue 10. According to an embodiment, the at least one input value 10may include at least one of a target resolution difference for the AIdown-scaler 1124 and the AI up-scaler 1144, a bitrate of the image data25, a bitrate type of the image data 25 (for example, a variable bitratetype, a constant bitrate type, or an average bitrate type), or a codectype for the transformation-based encoder 1126. The at least one inputvalue 10 may include a value pre-stored in the apparatus 20 or a valueinput from a user.

The configuration controller 1122 controls operations of the AIdown-scaler 1124 and the transformation-based encoder 1126, based on thereceived input value 10. According to an embodiment, the configurationcontroller 1122 obtains DNN setting information for the AI down-scaler1124 according to the received input value 10, and sets the AIdown-scaler 1124 with the obtained DNN setting information. According toan embodiment, the configuration controller 1122 may transmit thereceived input value 10 to the AI down-scaler 1124 and the AIdown-scaler 1124 may obtain the DNN setting information for performingAI down-scaling on the original image 105, based on the received inputvalue 10. According to an embodiment, the configuration controller 1122may provide, to the AI down-scaler 1124, additional information, forexample, color format (luminance component, chrominance component, redcomponent, green component, or blue component) information to which AIdown-scaling is applied and tone mapping information of a high dynamicrange (HDR), together with the input value 10, and the AI down-scaler1124 may obtain the DNN setting information considering the input value10 and the additional information. According to an embodiment, theconfiguration controller 1122 transmits at least a part of the receivedinput value 10 to the transformation-based encoder 1126 and thetransformation-based encoder 1126 performs first encoding on the firstimage 115 via a bitrate of a certain value, a bitrate of a certain type,and a certain codec.

The AI down-scaler 1124 receives the original image 105 and performs anoperation described with reference to at least one of FIG. 1, 7, 8, 9,or 10 to obtain the first image 115.

According to an embodiment, the AI data 30 is provided to the apparatus40. The AI data 30 may include at least one of resolution differenceinformation between the original image 105 and the first image 115, orinformation related to the first image 115. The resolution differenceinformation may be determined based on the target resolution differenceof the input value 10, and the information related to the first image115 may be determined based on at least one of a target bitrate, thebitrate type, or the codec type. According to an embodiment, the AI data30 may include parameters used during the AI up-scaling. The AI data 30may be provided from the AI down-scaler 1124 to the apparatus 40.

The image data 25 is obtained as the original image 105 is processed bythe transformation-based encoder 1126, and is transmitted to theapparatus 40. The transformation-based encoder 1126 may process thefirst image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8,VP9, or VA1.

A configuration controller 1142 controls an operation of the AIup-scaler 1144, based on the AI data 30. According to an embodiment, theconfiguration controller 1142 obtains the DNN setting information forthe AI up-scaler 1144 according to the received AI data 30, and sets theAI up-scaler 1144 with the obtained DNN setting information. Accordingto an embodiment, the configuration controller 1142 may transmit thereceived AI data 30 to the AI up-scaler 1144 and the AI up-scaler 1144may obtain the DNN setting information for performing AI up-scaling onthe second image 135, based on the AI data 30. According to anembodiment, the configuration controller 1142 may provide, to the AIup-scaler 1144, additional information, for example, the color format(luminance component, chrominance component, red component, greencomponent, or blue component) information to which AI up-scaling isapplied, and the tone mapping information of HDR, together with the AIdata 30, and the AI up-scaler 1144 may obtain the DNN settinginformation considering the AI data 30 and the additional information.According to an embodiment, the AI up-scaler 1144 may receive the AIdata 30 from the configuration controller 1142, receive at least one ofprediction mode information, motion information, or quantizationparameter information from the transformation-based decoder 1146, andobtain the DNN setting information based on the AI data 30 and at leastone of the prediction mode information, the motion information, and thequantization parameter information.

The transformation-based decoder 1146 may process the image data 25 toreconstruct the second image 135. The transformation-based decoder 1146may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4,HEVC, VC-1, VP8, VP9, or AV1.

The AI up-scaler 1144 may obtain the third image 145 by performing AIup-scaling on the second image 135 provided from thetransformation-based decoder 1146, based on the set DNN settinginformation.

The AI down-scaler 1124 may include a first DNN and the AI up-scaler1144 may include a second DNN, and according to an embodiment, DNNsetting information for the first DNN and second DNN are trainedaccording to the training method described with reference to FIGS. 9 and10.

FIG. 12 is a flowchart of an AI decoding method according to anembodiment.

In operation S910, the AI decoding apparatus 200 receives AI encodingdata including image data and AI data. The AI decoding apparatus 200 mayreceive the AI encoding data from the AI encoding apparatus 600 througha network. The AI decoding apparatus 200 may obtain the AI encoding datastored in a data storage medium.

In operation S920, the AI decoding apparatus 200 obtains the secondimage 135 based on the image data. In particular, the AI decodingapparatus 200 reconstructs the second image 135 corresponding to thefirst image 115 by decoding the image data based on an imagereconstruction method using frequency transform.

In operation S930, the AI decoding apparatus 200 obtains DNN settinginformation for performing AI up-scaling on the second image 135, fromamong a pre-stored plurality of pieces of DNN setting information.Because each of the plurality of pieces of DNN setting information isjointly optimized with each of a plurality of pieces of DNN settinginformation used to perform AI down-scaling on the original image 105,the DNN setting information enabling AI up-scaling to be performed onthe second image 135 according to an up-scaling target matching adown-scaling target of the original image 105 needs to be selected.

In operation S940, the AI decoding apparatus 200 obtains the third image145 obtained by performing AI up-scaling on the second image 135,through a second DNN operating with the DNN setting information obtainedin operation S930. The third image 145 may be output from the AIdecoding apparatus 200 and displayed through a display device or may bedisplayed after being post-processed.

When the DNN setting information is pre-set in the second DNN and theDNN setting information selected in operation S930 is different from thepre-set DNN setting information, the AI decoding apparatus 200 sets thesecond DNN to the selected DNN setting information.

FIG. 13 is a flowchart of an AI encoding method according to anembodiment.

In operation S1010, the AI encoding apparatus 600 obtains the firstimage 115 obtained by performing AI down-scaling the original image 105,through a first DNN.

The AI encoding apparatus 600 may determine a down-scaling target basedon a certain standard, and obtain DNN setting information correspondingto the down-scaling target from among a pre-stored plurality of piecesof DNN setting information. Also, the AI encoding apparatus 600 mayperform AI down-scaling on the original image 105 through the first DNNoperating according to the obtained DNN setting information.

In operation S1020, the AI encoding apparatus 600 obtains image data byperforming first encoding on the first image 115. In particular, the AIencoding apparatus 600 obtains the image data corresponding to the firstimage 115 by encoding the first image 115 based on an image compressionmethod using frequency transform.

In operation S1030, the AI encoding apparatus 600 transmits AI encodingdata including the image data and AI data including information relatedto AI down-scaling. The AI data includes information for selecting DNNsetting information of a second DNN for AI up-scaling of the secondimage 135. According to an embodiment, the AI encoding data may bestored in a data storage medium.

As described above, because the first DNN and the second DNN are jointlytrained, when the AI encoding apparatus 600 performs AI down-scaling onthe original image 105 to a particular down-scaling target, the AIdecoding apparatus 200 performs AI up-scaling on the second image 135 toan up-scaling target corresponding to the down-scaling target.

Accordingly, the AI data includes information enabling the AI decodingapparatus 200 to perform AI up-scaling on the second image 135 to theup-scaling target corresponding to the down-scaling target of theoriginal image 105. In particular, the AI data includes information usedto obtain DNN setting information corresponding to the up-scalingtarget.

Upon receiving the AI data, the AI decoding apparatus 200 is able toinfer or verify which DNN setting information is used by the AI encodingapparatus 600 to perform AI down-scaling on the original image 105, andaccordingly, may obtain DNN setting information corresponding to the DNNsetting information used to perform AI down-scaling, and perform AIup-scaling by using the obtained DNN setting information.

Meanwhile, the embodiments of the disclosure described above may bewritten as computer-executable programs or instructions that may bestored in a medium.

The medium may continuously store the computer-executable programs orinstructions, or temporarily store the computer-executable programs orinstructions for execution or downloading. Also, the medium may be anyone of various recording media or storage media in which a single pieceor plurality of pieces of hardware are combined, and the medium is notlimited to a medium directly connected to a computer system, but may bedistributed on a network. Examples of the medium include magnetic media,such as a hard disk, a floppy disk, and a magnetic tape, opticalrecording media, such as CD-ROM and DVD, magneto-optical media such as afloptical disk, and ROM, RAM, and a flash memory, which are configuredto store program instructions. Other examples of the medium includerecording media and storage media managed by application storesdistributing applications or by websites, servers, and the likesupplying or distributing other various types of software.

Meanwhile, a model related to the DNN described above may be implementedvia a software module. When the DNN model is implemented via a softwaremodule (for example, a program module including instructions), the DNNmodel may be stored in a computer-readable recording medium.

Also, the DNN model may be a part of the AI decoding apparatus 200 or AIencoding apparatus 600 described above by being integrated in a form ofa hardware chip. For example, the DNN model may be manufactured in aform of an dedicated hardware chip for AI, or may be manufactured as apart of an existing general-purpose processor (for example, CPU orapplication processor) or a graphic-dedicated processor (for exampleGPU).

Also, the DNN model may be provided in a form of downloadable software.A computer program product may include a product (for example, adownloadable application) in a form of a software program electronicallydistributed through a manufacturer or an electronic market. Forelectronic distribution, at least a part of the software program may bestored in a storage medium or may be temporarily generated. In thiscase, the storage medium may be a server of the manufacturer orelectronic market, or a storage medium of a relay server.

While one or more embodiments of the disclosure have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope as defined by thefollowing claims.

What is claimed is:
 1. An electronic device for displaying an image byusing artificial intelligence (AI), the electronic device comprising: adisplay; and one or more processors, when executing one or moreinstructions stored in the electronic device, configured to: receive AIdata related to AI down-scaling an original image to a first imagethrough a down-scaling neural network (NN) based on first NN settinginformation, and an image data generated through an encoding on thefirst image, the first NN setting information being selected from aplurality of first NN setting information, and the AI data comprising abitrate corresponding to the image data; obtain a second image bydecoding the image data; select, based on the AI data, second NN settinginformation paired with the selected first NN setting information from aplurality of second NN setting information; obtain, by an up-scaling NN,a third image by performing AI up-scaling on the obtained second image,the up-scaling NN being set with the selected second NN settinginformation; and provide, on the display, the obtained third image,wherein the plurality of first NN setting information and the pluralityof second NN setting information are paired with each other.
 2. Theelectronic device of claim 1, wherein the AI data comprises informationrelated to the first image, wherein the one or more processors, whenexecuting the one or more instructions, are further configured to selectthe second NN setting information mapped to the information related tothe first image, based on a mapping relationship between a plurality ofimage-related information and the plurality of second NN settinginformation.
 3. The electronic device of claim 1, wherein the AI datafurther comprises at least one of a quantization parameter regarding theimage data, a resolution of the first image, or a codec type used toencode the first image.
 4. The electronic device of claim 1, wherein theselected second NN setting information comprises neural networkparameters for at least one convolution layer in the up-scaling NN.
 5. Aserver for providing an image by using an artificial intelligence (AI),the server comprising: one or more processors configured to execute oneor more instructions stored in the server to: select first neuralnetwork (NN) setting information from a plurality of first NN settinginformation; obtain, by a down-scaling NN, a first image by performingAI down-scaling on an original image, the down-scaling NN being set withthe selected first NN setting information; encode the first image toobtain image data; and provide an electronic device with the image dataand AI data related to the AI down-scaling, the AI data comprising abitrate corresponding to the image data and being used to select secondNN setting information paired with the selected first NN settinginformation from a plurality of second NN setting information for AIup-scaling, wherein the plurality of first NN setting information andthe plurality of second NN setting information are paired with eachother, and the AI up-scaling is performed by an up-scaling NN set withthe selected second NN setting information.
 6. The server of claim 5,wherein the one or more processors is configured to select the first NNsetting information from the plurality of first NN setting informationbased on a compression ratio input from a user.
 7. The server of claim5, wherein the one or more processors is configured to select the firstNN setting information from the plurality of first NN settinginformation based on compression history information.
 8. The server ofclaim 5, wherein the one or more processors is configured to select thefirst NN setting information from the plurality of first NN settinginformation based on at least one of a resolution of the original imageor a type of the original image.
 9. The server of claim 5, wherein theAI data further comprises at least one of a quantization parameterregarding the image data, a resolution of the first image, or a codectype used to encode the first image.
 10. A non-transitorycomputer-readable recording medium having recorded thereon instructions,which when executed by an electronic device, cause the electronic deviceto perform first operations comprising: receiving artificialintelligence (AI) data related to AI down-scaling an original image to afirst image through a down-scaling neural network (NN) based on first NNsetting information, and image data corresponding to an encoding resulton the first image, the first NN setting information being selected froma plurality of first NN setting information, and the AI data comprisinga bitrate corresponding to the image data; decoding the image data toobtain a second image; and providing, on a display of the electronicdevice, a third image that is obtained by performing AI up-scaling onthe second image, and which when executed by the electronic device,causes the electronic device to perform second operations comprising:selecting, based on the AI data, second NN setting information pairedwith the selected first NN setting information, from a plurality ofsecond NN setting information for the AI up-scaling; and obtaining, byan up-scaling NN, the third image by performing the AI up-scaling on theobtained second image, the up-scaling NN being set with the selectedsecond NN setting information, wherein the plurality of first NN settinginformation and the plurality of second NN setting information arepaired with each other.